Systems and methods for training a non-blind image deblurring module

ABSTRACT

Methods and systems for training a non-blind deblurring module are disclosed. Unblurred test images and blurred test images are received, wherein each of the blurred test images is related to a corresponding one of the unblurred test images by a blur kernel term and a noise term. A regularized deconvolution sub-module and a convolutional neural network are jointly trained by adjusting a regularization parameter of a regularized deconvolution function and weights of a convolution neural network in order to minimize a cost function representative of a difference between each deblurred output image and a corresponding one of the unblurred test images.

TECHNICAL FIELD

The present disclosure generally relates to non-blind image deblurring,and more particularly relates to methods and systems for training anon-blind image deblurring module.

An image captured by a camera may be blurred for a variety of reasons.For example, the camera may have been moving or shaking during imagecapture. Image blur may also be caused by optical aberrations. Chromaticblur is also prevalent by which different wavelengths are refracted todifferent extents. Non-blind deconvolution techniques are known by whicha blurred input image is processed in order to obtain a clearer,deblurred, output image. According to such deconvolution techniques, ablur kernel is utilized to transform the blurred input image to adeblurred output image. The blur kernel may be determined from a pointspread function representing the nature of anticipated blurring effects.In the case of a camera mounted to a moving vehicle, the point spreadfunction is derived based on knowledge of the vehicle motion and thedeblurring kernel is determined based on the point spread function. Thatis, a blur source is often well known in imaging and the blurringprocess can be modeled quite well using a point spread function (PSF)that is directly measured or derived from knowledge of the physics ofblurring.

Most blurred images include noise as well as blurring. The noise furthercomplicates the deblurring problem. Classical techniques usually removethe blur but add other artifacts. Brute-force direct application of DeepNeural Nets may be successful but only for relatively weak blur.Further, since the deconvolution method is largely changed according toinfluences of the noise and changes of the blur kernel, methods usingdeep learning techniques have been limited in accuracy and calculationspeed.

Image deblurring can often be reduced to deconvolution using the blurkernel. Deconvolution is an ill-posed inverse problem. As such, itshould be regularized. Regularization parameters add stability to thesolution. However, optimizing the regularization parameters is achallenging task. If there is too much stability, a blurry output mayresult, whilst too little stability may result in noise amplification.

Accordingly, it is desirable to provide systems and methods that providenon-blind image deblurring that operate efficiently through the use ofregularized deconvolution techniques, whereby a regularization parameterhas been efficiently and optimally selected to improve deblurred imageoutputs. Furthermore, other desirable features and characteristics ofthe present invention will be apparent from the subsequent detaileddescription and the appended claims, taken in conjunction with theaccompanying drawings and the foregoing technical field and background.

SUMMARY

In accordance with an exemplary embodiment, a method of training anon-blind deblurring module is provided. The non-blind deblurring moduleincludes a regularized deconvolution sub-module and a convolutionalneural network sub-module. The regularized deconvolution sub-module isconfigured to perform a regularized deconvolution function on a blurredinput image to produce a deconvolved image potentially having imageartifacts. The convolutional neural network sub-module is configured toreceive the deconvolved image as an input to a convolutional neuralnetwork and to remove image artifacts, thereby providing a deblurredoutput image. The method includes receiving, via at least one processor,unblurred test images and blurred test images. Each of the blurred testimages is related to a corresponding one of the unblurred test images bya blur kernel term and a noise term. The method includes jointlytraining, via the at least one processor, the regularized deconvolutionsub-module and the convolutional neural network. Joint training includesadjusting a regularization parameter of the regularized deconvolutionfunction and weights of the convolution neural network in order tominimize a cost function representative of a difference between eachdeblurred output image and a corresponding one of the unblurred testimages, thereby providing a trained regularization parameter, trainedweights and a trained non-blind deblurring module. The method furtherincludes receiving, via the at least one processor, a blurred inputimage from an imaging device, deblurring, via the at least oneprocessor, the blurred input image using the trained non-blinddeblurring module, and outputting, via at least one processor, adeblurred output image.

In embodiments, the deconvolution function is a Wiener deconvolutionfunction. In embodiments, the deconvolution function is aTikhonov-regularized deconvolution function.

In embodiments, the method includes using the trained non-blinddeblurring module to deblur the blurred input image, thereby producingthe deblurred output image. The regularized deconvolution sub-moduleperforms the regularized deconvolution function on the blurred inputimage to produce a deconvolved image potentially having image artifacts,the regularized deconvolution function including the trainedregularization parameter, and the convolutional neural networksub-module processes the deconvolved image through the convolutionalneural network to remove image artifacts, the convolution neural networkincluding the trained weights.

In embodiments, the convolutional neural network outputs residuals andthe trained non-blind deblurring module adds the residuals to thedeconvolved image, thereby producing the deblurred output image.

In embodiments, the method includes adjusting, via the at least oneprocessor, the regularization parameter and the weights using a backpropagation algorithm. In embodiments, the back propagation algorithmadjusts the regularization parameter based on gradients that have beenfed back from the CNN and a derivative of the deconvolved imagepotentially having image artifacts with respect to the regularizationparameter.

In embodiments, the at least one processor receives the unblurred testimages and artificially generates the blurred test images using a blurkernel function and a noise function on the unblurred test images.

In embodiments, the blurred input image is received from the imagingdevice mounted to a vehicle.

In embodiments, the vehicle includes a vehicle controller. The methodincludes the vehicle controller controlling at least one vehiclefunction based on the deblurred output image.

In accordance with another exemplary embodiment, a system for training anon-blind deblurring module is provided. The system includes a non-blinddeblurring module including a regularized deconvolution sub-module and aconvolutional neural network sub-module. The regularized deconvolutionsub-module is configured to perform a regularized deconvolution functionon a blurred input image to produce a deconvolved image potentiallyhaving image artifacts. The convolutional neural network sub-module isconfigured to receive the deconvolved image as an input to aconvolutional neural network and to remove image artifacts, therebyproviding a deblurred output image. The system includes at least oneprocessor configured to execute program instructions. The programinstructions are configured to cause the at least one processor toreceive unblurred test images and blurred test images. Each of theblurred test images is related to a corresponding one of the unblurredtest images by a blur kernel term and a noise term. The programinstructions are configured to cause the at least one processor tojointly train the regularized deconvolution sub-module and theconvolutional neural network by adjusting a regularization parameter ofthe regularized deconvolution function and weights of the convolutionneural network in order to minimize a cost function. The cost functionis representative of a difference between each deblurred output imageand a corresponding one of the unblurred test images, thereby providinga trained regularization parameter, trained weights and a trainednon-blind deblurring module. The program instructions further causingthe at least one processor to: receive a blurred input image from theimaging device, deblur the blurred input image using the trainednon-blind deblurring module, and output a deblurred output image.

In embodiments, the deconvolution function is a Wiener deconvolutionfunction. In embodiments, the deconvolution function is aTikhonov-regularized deconvolution function.

In embodiments, the trained non-blind deblurring module is configured todeblur the blurred input image, thereby producing the deblurred outputimage. The regularized deconvolution sub-module is configured to performthe regularized deconvolution function on the blurred input image toproduce a deconvolved image potentially having image artifacts using thetrained regularization parameter. The convolutional neural networksub-module is configured to process the deconvolved image through theconvolutional neural network to remove image artifacts using the trainedweights.

In embodiments, the convolutional neural network is configured to outputresiduals and the trained non-blind deblurring module is configured toadd the residuals to the deconvolved image, thereby producing thedeblurred output image.

The program instructions are configured to cause the at least oneprocessor to adjust the regularization parameter and the weights using aback propagation algorithm. In embodiments, the program instructions areconfigured to cause the at least one processor to adjust theregularization parameter based on gradients that have been fed back fromthe CNN and a derivative of the deconvolved image potentially havingimage artifacts with respect to the regularization parameter.

In embodiments, the program instructions are configured to cause the atleast one processor to receive the unblurred test images andartificially generate the blurred test images using a blur kernelfunction and a noise function on the unblurred test images.

In embodiments, the system includes a vehicle. The vehicle includes acamera and the non-blind deblurring module. The non-blind deblurringmodule is configured to receive the blurred input image from the camera.

In embodiments, the vehicle includes a control module configured tocontrol at least one vehicle function based on the deblurred outputimage.

DESCRIPTION OF THE DRAWINGS

The present disclosure will hereinafter be described in conjunction withthe following drawing figures, wherein like numerals denote likeelements, and wherein:

FIG. 1 is a functional block diagram of a system for non-blinddeblurring, in accordance with an exemplary embodiment;

FIG. 2 is a functional block diagram of data processes in a regularizeddeconvolution sub-module, in accordance with an exemplary embodiment;

FIG. 3 is a functional block diagram of data processes in anotherregularized deconvolution sub-module, in accordance with an exemplaryembodiment;

FIG. 4 is a functional block diagram of a system for training anon-blind deblurring module, in accordance with an exemplary embodiment;

FIG. 5 is a functional block diagram representing data transformationprocessed in a non-blind deblurring module, in accordance with anexemplary embodiment;

FIG. 6 is a flowchart of a method of training and using a non-blinddeblurring module, in accordance with an exemplary embodiment; and

FIG. 7 shows blurred input images, deconvolved images and deblurredoutput images according to use of the system of FIG. 1, in accordancewith an exemplary embodiment.

DETAILED DESCRIPTION

The following detailed description is merely exemplary in nature and isnot intended to limit the disclosure or the application and usesthereof. Furthermore, there is no intention to be bound by any theorypresented in the preceding background or the following detaileddescription.

As used herein, the term module refers to any hardware, software,firmware, electronic control component, processing logic, and/orprocessor device, individually or in any combination, including withoutlimitation: application specific integrated circuit (ASIC), anelectronic circuit, a processor (shared, dedicated, or group) and memorythat executes one or more software or firmware programs, a combinationallogic circuit, and/or other suitable components that provide thedescribed functionality.

Embodiments of the present disclosure may be described herein in termsof functional and/or logical block components and various processingsteps. It should be appreciated that such block components may berealized by any number of hardware, software, and/or firmware componentsconfigured to perform the specified functions. For example, anembodiment of the present disclosure may employ various integratedcircuit components, e.g., memory elements, digital signal processingelements, logic elements, look-up tables, or the like, which may carryout a variety of functions under the control of one or moremicroprocessors or other control devices. In addition, those skilled inthe art will appreciate that embodiments of the present disclosure maybe practiced in conjunction with any number of systems, and that thesystems described herein is merely exemplary embodiments of the presentdisclosure.

For the sake of brevity, conventional techniques related to signalprocessing, data transmission, signaling, control, and other functionalaspects of the systems (and the individual operating components of thesystems) may not be described in detail herein. Furthermore, theconnecting lines shown in the various figures contained herein areintended to represent example functional relationships and/or physicalcouplings between the various elements. It should be noted that manyalternative or additional functional relationships or physicalconnections may be present in an embodiment of the present disclosure.

Described herein are systems and methods for non-blind image deblurringbased on a hybrid system using a classical regularized deconvolutionmodule and a Convolutional Neural Network (CNN). Parameters of theregularization module are determined together with CNN weights by ajoint training procedure. Training data includes any relevant imagedatabase and system-generated blur and noise images. The systems andmethods disclosed herein have been found to produce artifact free imagedeblurring even for almost arbitrarily strong blur in the input images.The system and methods disclosed herein are relevant for multiplesensing scenarios—high exposure-time motion blur, chromatic blur, etc.

FIG. 1 illustrates a system 10 for non-blind image deblurring. Thesystem 10 includes a vehicle 12, an imaging device 14 mounted to thevehicle 12, vehicle sensors 16, a vehicle controller, a vehiclecontroller 18 and a usage processing system 26. Usage processing system26 includes a non-blind deblurring module 34 executed by a processor 70and computer program instructions 74 stored on memory 72. Inembodiments, imaging device 14 is configured to capture images asblurred input images 24, which are deblurred using a regularizeddeconvolution function 44 and a convolutional neural network, CNN 42included in non-blind deblurring module to thereby produce deblurredoutput images 20. The deblurred output images are used by vehiclecontroller 18 to control one or more functions of the vehicle 12.

The system 10 is shown in the context of (e.g. included within) thevehicle 12, specifically an automobile. The system 10, however, isuseful in other vehicular contexts such as aircraft, sea vessels, etc.The system 10 is applicable outside of vehicular contexts including anyelectronic device, such as a mobile telephone, a camera, and a tabletdevice, that captures an image subject to blurring. The presentdisclosure is particularly, but not exclusively, relevant to blurring asa result of motion (which will often occur in vehicular contextsparticularly at night or other times when exposure times are extended).The system 10 is useful with respect to other blurring sources thanmotion such as those caused by optical aberrations or chromaticfiltering.

In various embodiments, the vehicle 12 is an autonomous vehicle and thesystem 10 is incorporated into the autonomous vehicle 12. However, thesystem 10 is useful in any kind of vehicle (autonomous or otherwise)that includes the imaging device 14 that produces images subject toblurring. The autonomous vehicle 12 is, for example, a vehicle that isautomatically controlled to carry passengers from one location toanother. The vehicle 12 is depicted in the illustrated embodiment as apassenger car, but it should be appreciated that any other vehicleincluding motorcycles, trucks, sport utility vehicles (SUVs),recreational vehicles (RVs), marine vessels, aircraft, etc., can also beused. In an exemplary embodiment, the autonomous vehicle 12 is aso-called Level Four or Level Five automation system. A Level Foursystem indicates “high automation”, referring to the drivingmode-specific performance by an automated driving system of all aspectsof the dynamic driving task, even if a human driver does not respondappropriately to a request to intervene. A Level Five system indicates“full automation”, referring to the full-time performance by anautomated driving system of all aspects of the dynamic driving taskunder all roadway and environmental conditions that can be managed by ahuman driver.

In embodiments, the vehicle 12 includes the vehicle controller 18configured to control one or more vehicular functions based on imagesfrom the imaging device 14. The vehicle controller 18 may include one ormore advanced driver-assistance systems configured to provide electronicdriver assistance based on images from the imaging device 14. Thevehicle controller 18 may include an autonomous driver orsemi-autonomous driver configured to control the vehicle 12 through oneor more actuation systems (e.g. propulsion, braking and steeringsystems) based on images input from the imaging device 14. In all suchembodiments, better deblurred input images will allow the vehiclecontroller 18 to more safely control the vehicle 12.

In accordance with various embodiments, system 10 includes the imagingdevice 14 (e.g. a front, rear or side mounted camera), vehicle sensors16, the vehicle controller 18 and the usage processing system 26. Theusage processing system 26 is configured, by programming instructions 74executing on a processor 70 (as described further below), to receive ablurred input image 24 from the imaging device 14 and to perform theregularized deconvolution function 44 on the blurred input image thatrelies upon a regularization parameter. The resulting deconvolved image40 is passed through the convolutional neural network (CNN) 42 in orderto remove any image artifacts that have arisen from the deconvolutionfunction 44. According to the present disclosure, weights of the CNN 42and the regularization parameter of the deconvolution function 44 arejointly trained. That is, the regularization parameter is part of a backpropagation chain when training the CNN 42. The jointly trained networklayers include layers of CNN 42 and a deconvolution layer.

Continuing to refer to FIG. 1, the usage processing system 26 includesthe at least one processor 70, the memory 72, and the like. Theprocessor 70 may execute the program instructions 74 stored in thememory 72. The processor 70 may refer to a central processing unit(CPU), a graphics processing unit (GPU), or a dedicated processor onwhich the methods and functions according to the present disclosure areperformed. The memory 72 may be composed of a volatile storage mediumand/or a non-volatile storage medium. For example, the memory 72 may becomprised of a read only memory (ROM) and/or a random access memory(RAM). The memory 72 stores at least one instruction executed by theprocessor 70 in order to implement blocks, modules and method stepsdescribed herein. Although modules 28, 30 and 34 are shown separatelyfrom the processor 70, the memory 72 and the programming instructions74, this is purely for visualization. In reality, the modules 28, 30 and34 are embodied by programming instructions 74 stored on the memory 72and executable by one or more processors 70 of the usage processingsystem 26.

The imaging device 14 is any suitable camera or video device thatproduces images. Images are presumed to include blurring (and thus arelabelled blurred input images 24) for the purposes of the presentdisclosure as a result of motion blurring or other types of blursources. The imaging device 14 may be a color imaging device or agrayscale imaging device. The imaging device 14 may operate in visibleand/or infrared spectrums. The imaging device 14 may produce one, two orthree-dimensional (1D, 2D or 3D) images serving as the blurred inputimages 24.

The vehicle sensors 16 include a variety of sensors used by vehiclecontroller 18 in order to control operation of the vehicle 12. Ofparticular relevance to the present disclosure are speed (e.g. wheelspeed sensors), acceleration (accelerometers and gyroscopes) and othervehicle sensors 16 that provide data 22 representing sensed parametersof vehicle motion. Motion parameters data 22 is used by the usageprocessing system 26 in order to determine a point spread function(PSF), as described further herein. The point spread function asdetermined by the usage processing system is used in deconvolutionfunction 44 to deblur the blurred input image 24. Although the presentdisclosure is described, in the most part, in terms of a PSF that isdynamically determined based on motion parameters data 22, the disclosedsystems and methods are applicable in other applications whenever theblurring model is known. Sometimes blurring is inherent in the imagingdevice 14 itself and the imaging device can be calibrated to measure thePSF directly without reference to external sensed data. The presentdisclosure finds use in those applications as well.

Blurring of an input image can be represented mathematically by thefollowing equation:

I _(B) =I×K _(B)  (equation 1)

where I_(B) is a blurred input image 24, I is an unknown, unblurredimage corresponding to the blurred input image 24 and K_(B) is a matrixor blur kernel that models a point spread function (PSF) describing thenature of the blurring in the blurred input image 24. Since the presentdisclosure is concerned with non-blind deblurring, it is assumed thatthe PSF is known and thus the blur kernel K_(B) can be derived from thePSF. PSFs for all manner of blurring are known in the art includingblurring caused by motion of imaging device 14 during exposure. Intheory, the inverse of the blur kernel or matrix (the inverse beingrepresented by Kg′) can be multiplied with the blurred input image 24 toresolve the unblurred image I. However, noise in the blurred input image24 makes such a straightforward deconvolution impractical. The noisecomponent will be amplified during the deconvolution process in anuncontrolled way, likely leading to deconvolved images that are lessclear (or more blurred) than the original blurred input image 24. Onesolution to this noise amplification problem is to use a regularizedinverse of the blur kernel to deblur the blurred input image 24.

Such regularized deconvolution functions are known in the art and twoexamples will be provided in the following. Regularized deconvolutionfunctions rely on a regularization parameter λ to mitigate the effectsof noise. The regularization parameter λ has a large impact on qualityof deblurring. If the regularization parameter λ is too low, then noisecan have a significant impact on the output image. If the regularizationparameter λ is too high, then blur is magnified in the output image.According to the present disclosure, the regularization parameter λ isdetermined as part of a joint training process for CNN weights of CNN42.

Referring to FIG. 1, the regularization deconvolution sub-module 36receives blurred input image 24 and operates the regularizeddeconvolution function 44 thereon. The regularized deconvolutionfunction 44 includes an inverse regularized blur kernel or matrix thatis determined as a function of trained regularization parameter 2. Usageprocessing system 26 includes a point spread function module 28, whichreceives motion parameters data 22 including, at least speed andoptionally acceleration data, to determine point spread function (PSF)data 30 representing a PSF function. Psf data 31 is variable dependingupon vehicle motion (e.g. the faster the vehicle, the greater the spreador blur defined by the PSF) and camera data 76 representing relevantcamera parameters obtained from imaging device 14 (e.g. exposure time).Psf data 31 is determined by point spread function module 28, whichincludes a modelling function for determining anticipated PSF based onmotion parameters data 22 and camera data 76. Blur kernel determinationmodule 30 transforms the PSF defined in Psf data 31 into matrix form andoutputs corresponding blur kernel data 32. In cases where motion inducedblur is not the source of blur, then determining the PSF in this way isnot necessary.

Regularized deconvolution sub-module 36 receives blur kernel data 32representing blur kernel K_(B) and makes use thereof in carrying out aregularized deconvolution on blurred input image 24 in order to generatedeconvolved image 40. It should be appreciated that many varieties ofPSFs and the determination thereof are available depending upon thenature of the source of blur. Although the present disclosure is largelydescribed in terms of motion blur and in association with vehicleapplications, other blur sources, and thus other ways of determining thePSF, can be incorporated in the present disclosure depending on theapplication. Thus, point spread function module 28 is not necessarilydependent on motion parameters data 22 or camera data 76 (specificallyexposure time). For example, when imaging over a broad range ofwavelengths (e.g. more than one color), the combined image will often beblurred because each of a plurality of wavelength bands refract to adifferent extent by optics of the imaging device 14. A point spreadfunction can be defined to reverse such chromatic blurring.

FIG. 2 illustrates an exemplary data flow diagram for a regularizeddeconvolution function 44 used by the regularized deconvolutionsub-module 36 in the case of 1D motion blur. The regularizeddeconvolution function 44 is, in this case, a Tikhonov regularizeddeconvolution function. Following FIG. 2, the PSF module 28 generatesPsf data 31 representing PSF, which is transformed into the blur kernelK_(B) defined by the blur kernel data 32. The blur kernel is subject toa singular-value decomposition (SVD) in order to generate USVdecomposition matrices according to equation 2:

K _(B) =USV ^(T)  (equation 2)

The inverse of the regularized blur kernel is found to be:

[K _(B)]_(REG) ⁻¹ =VS(S ²+λ² I)⁻¹ U ^(T)  (equation 3)

where I (unblurred version of blurred input image 24) is:

I≃I _(B)[K _(B)]_(REG) ⁻¹  (equation 4)

and I_(B) is the blurred input image 24.

Continuing to refer to FIG. 2, the regularized deconvolution function 44is made up by inverse decomposed matrices 80, 82, 84 as a result of anSVD and inversion operation on the blur kernel K_(B). The matrix 82 is afunction of the trained regularization parameter λ 78 and is of the formS(S²+λ²I)⁻¹. The blurred input image 24 is multiplied by decompositionmatrices 80, 82, 84 as part of the regularized Tikhonov deconvolutionfunction 44 to thereby provide a deconvolved image 40.

FIG. 3 illustrates an exemplary data flow diagram for a regularizeddeconvolution function 44 used by the regularized deconvolutionsub-module 36 in the case of 2D motion blur. The regularizeddeconvolution function 44 is, in this case, a Wiener regularizeddeconvolution function. The Wiener regularized deconvolution function isdefined by the following equations:

$\begin{matrix}{{{{\overset{\sim}{I}}_{B}( \overset{arrow}{k} )} = {{\overset{\sim}{I}( \overset{arrow}{k} )}{{\overset{\sim}{K}}_{B}( \overset{arrow}{k} )}}},} & ( {{equation}\mspace{14mu} 6} ) \\{{{\overset{\sim}{I}( \overset{arrow}{k} )} = {{FFT}(I)}},{and}} & ( {{equation}\mspace{14mu} 7} ) \\{{\overset{\sim}{I}( \overset{arrow}{k} )} = {{{{\overset{\sim}{I}}_{B}( \overset{arrow}{k} )}\frac{{\overset{\sim}{K}}_{B}^{+}( \overset{arrow}{k} )}{{{{\overset{\sim}{K}}_{B}( \overset{arrow}{k} )}}^{2} + \lambda^{2}}} = {{{\overset{\sim}{I}}_{B}( \overset{arrow}{k} )}{{\overset{\sim}{K}}_{B}^{- 1}( {\overset{arrow}{k},\lambda} )}}}} & ( {{equation}\mspace{14mu} 8} )\end{matrix}$

Following FIG. 3, the PSF module 28 generates Psf data 31 representingPSF, which is transformed into the blur kernel K_(B) defined by blurkernel data 32. The blur kernel K_(B) is subject to a first Fast FourierTransform (FFT) operation 88 as required by equation 7. Similarly, theblurred input image 24 is subject to a second FFT operation 90 and thenoperated upon by an inverse of the regularized blur kernel at operation92, which is represented by {tilde over (K)}_(B) ⁻¹({right arrow over(k)}, λ) and is a function of the trained regularization parameter λ 78.The Wiener regularized deconvolution function further includes aninverse FFT operation 94 for the output of the operation 92. In thisway, the deconvolved image 40 is output by the regularized deconvolutionsub-module 36.

The deconvolved image 40 will, generally, have artifacts as an inherentresult of the regularized deconvolution process. For that reason, thenon-blind deblurring module 34 includes the CNN sub-module 38 having theCNN 42 configured, through training, to remove any image artifacts inthe deconvolved image 40. In one embodiment, the CNN 42 is a residualU-net. The CNN 42 is trained to remove artifacts and, at the same time,the regularized deconvolution sub-module 36 is trained to determine anoptimal regularization parameter 2.

FIG. 4 provides a functional block diagram of a training processingsystem 50 for training the non-blind deblurring module 34, in accordancewith an exemplary embodiment. Training processing system 50 includes animage database 56 serving as a source of unblurred test images 60. Apoint spread function generation module 52 generates artificial PSFsembodied in artificial PSF data 54. Artificial PSFs are generated so asto be similar to PSFs that would occur during usage of the non-blinddeblurring module 34. For example, in the case of vehicularapplications, randomized PSFs are generated representing 1D or 2Ddeblurring based on exposure times and motion parameters that would beencountered during operation of the vehicle 12. A noise generationmodule 62 provides artificial noise data 64 representative of noise tobe applied to the unblurred test image 60. A gaussian function may beutilized by the noise generation module 62. The blurred image generationmodule 58 receives unblurred the test image 60, the artificial PSF data54 and the artificial noise data 64 and generates a blurred test image66 that corresponds to the unblurred test image 60 but includes blurringbased on the artificial PSF data 54 and noise based on the artificialnoise data 64.

The training processing system 50 includes a processor 91 and memory 93storing programming instructions 95. The processor 91 executes theprogramming instructions 95 in order to instigate a training process, aswill be described, and the various modules of the training processingsystem 50. During the training process, the regularized deconvolutionsub-module 36 is configured to receive the blurred test image 66 and todeconvolve (as described elsewhere herein) the blurred test image 66into a sharp, but potentially artifact ridden, deconvolved image 40.Exemplary artifacts include unpleasant ringing artifacts that appearnear strong edges. The CNN sub-module 38 is configured to receive thedeconvolved image and to generate output residuals 48 after passing thedeconvolved image through layers of the CNN 42. The residuals 48 arerepresentative of image artifacts after the CNN 42 has been trained. Theresiduals 48 and the deconvolved image 40 are combined at summingfunction 46 in order to produce a deblurred output image 20. Thetraining process, executed by training processing system 50, adjustsweights of the CNN 42 and the regularization parameter λ in order tomake the deblurred output image 20 match as closely as possible to theunblurred test image 60. That is, a cost function module 96 implements acost function 100 to generate cost data 98 representative of adifference between the deblurred output image 20 and the unblurred testimage 60.

The training processing system 50 jointly adjusts weights of the CNN 42and the regularization parameter λ of the regularized deconvolutionsub-module 36 in a repeating process in order to minimize the cost data98. In embodiments, the training processing system 50 utilizes a backpropagation algorithm. The back propagation algorithm computes thegradient of the cost function 100 with respect to the weights of the CNN42 and the gradient of the cost function 100 with respect to theregularization parameter λ for the unblurred test image 60 and thedeblurred output image 20. Such gradient methods jointly train layers ofthe CNN 42 and a regularized deconvolution layer and update weights andthe regularization parameter λ to minimize the cost function 100 (e.g.to minimize cost value defined by the cost data 98). The backpropagationalgorithm computes the gradient of the cost function 100 with respect toeach weight and the regularization parameter λ by the chain rule,computing the gradient one layer at a time, iterating backward from thelast layer of the CNN 42 (closest to the output) to the first layer ofthe CNN 42 and ending with the regularized deconvolution layer.

A layer-based conception of the training of the non-blind deblurringmodule 34 is shown in FIG. 5. The blurred test image 66 is passedthrough a regularized deconvolution layer 102 to produce a deconvolvedimage 40 that is subsequently passed through layers of the CNN 42 toproduce residuals 48 that are added to the deconvolved image 40 toprovide the deblurred output image 20. According to the presentdisclosure, the CNN layers 104 and the regularized deconvolution layer102 are trained by adjusting weights of the CNN layers 104 and theregularization parameter λ of the regularized deconvolution layer 102during the same back propagation process.

Within a standard gradient-based cost function optimization scheme (suchas SGD optimization), the update relation for any neural network (NN)parameter θ (from step n−1 to step n) is:

$\begin{matrix}{\theta_{n} = {\theta_{n - 1} - {\eta\frac{\partial L}{\partial\theta}}}} & ( {{equation}\mspace{14mu} 9} )\end{matrix}$

Where L is the cost function of the neural network and η is the learningrate. Typically, since the dependence of the cost function on any neuralnetwork parameter θ is manifested through a chain-like dependencescheme, the required gradients are calculated by a backpropagationalgorithm.

Specifically, if

L=g ₀(g ₁(g ₂( . . . (g _(N)(θ)))))  (equation 10)

then

$\begin{matrix}{\frac{\partial L}{\partial\theta} = {\frac{\partial L}{\partial g_{N}}\frac{\partial g_{N}}{\partial\theta}}} & ( {{equation}\mspace{14mu} 11} )\end{matrix}$

As regards the present disclosure, the overall cost, calculated by costfunction module 96, as a function of regularization parameter λ is:

L=L(J _(DB)(λ))  (equation 12)

That is, the cost function 100 is a function of deconvolved image J_(DB)40, which itself is a function of the regularization parameter λ. Inorder to determine the derivative of the cost L with respect toregularization parameter λ as required by equation 9, there is:

$\begin{matrix}{{\frac{\partial L}{\partial\lambda} = \frac{\partial L}{\partial J_{DB}}}\frac{\partial J_{DB}}{\partial\lambda}} & ( {{equation}\mspace{14mu} 13} )\end{matrix}$

$\frac{\partial L}{\partial J_{DB}}$

represents back propagation from CNN 42 and represents the accumulationof layer by layer derivatives as a result of back propagationcalculations through the CNN layers 104. The input gradients

$\frac{\partial L}{\partial J_{DB}}$

are fed back from the CNN 42 and represent the change in the costfunction 100 with respect to the change in the deconvolved image J_(DB).This term is multiplied by

$\frac{\partial J_{DB}}{\partial\lambda},$

which represents a change in the deconvolved image J_(DB) 40 withrespect to change in the regularization parameter λ.

In the case of Tikhonov regularized deconvolution,

$\begin{matrix}{{\frac{\partial L}{\partial\lambda} = {{\frac{\partial L}{\partial J_{DB}}\frac{\partial J_{DB}}{\partial\lambda}} = {{- ( \frac{\partial L}{\partial J_{DB}} )}J_{B}U\frac{2S\lambda}{( {S^{2} + {\lambda^{2}I}} )^{2}}V^{T}}}},} & ( {{equation}\mspace{14mu} 14} )\end{matrix}$

which can be derived from equation 3. The update scheme for theregularization parameter will be:

$\begin{matrix}{{\lambda_{n} = {\lambda_{n - 1} + {{\eta( \frac{\partial L}{\partial J_{DB}} )}J_{B}U\frac{2S\lambda}{( {S^{2} + {\lambda^{2}I}} )^{2}}V^{T}}}},} & ( {{equation}\mspace{14mu} 15} )\end{matrix}$

which can be derived from a combination of equations 9 and 14.

In the case of regularized Wiener deconvolution,

$\begin{matrix}{\lambda_{n} = {\lambda_{n - 1} + {{\eta( \frac{\partial L}{\partial J_{DB}} )}{{{IFFT}\lbrack {{J_{B}(k)}\frac{2\lambda\;{K^{*}(k)}}{( {{{K(k)}}^{2} + \lambda^{2}} )^{2}}} \rbrack}.}}}} & ( {{equation}\mspace{14mu} 16} )\end{matrix}$

For both of equations 15 and 16, gradients obtained through backpropagation through the CNN 42 allow calculation of

$\frac{\partial L}{\partial J_{DB}},$

which is multiplied by change in the deconvolved image 40 with respectto change in the regularization parameter, thereby providing a value forthe regularization parameter for the current step or iteration of thetraining processing system 50. Training processing system 50 repeatssuch steps in order to minimize the cost function 100 with respect toboth CNN weights and the regularization parameter. Training processingsystem 26 outputs the non-blind deblurring module 34 having trained CNNweights 106 loaded in the CNN 42 and a trained regularization parameter78 loaded in the deconvolution function 44.

FIG. 6 is a flowchart of a method 200 for training and using non-blinddeblurring module 34, in accordance with an exemplary embodiment. Themethod 200 can be implemented in connection with the vehicle 12 of FIG.1 and the usage and training processing systems 26, 50 of FIGS. 1 and 4,in accordance with an exemplary embodiment.

Method 200 includes step 210 of receiving the blurred input image 24. Inusage, the blurred input image 24 is received from the imaging device14, which may be associated with the vehicle 12 but may also beassociated with another device or apparatus. In training, the blurredinput image 66 may be artificially generated by the blurred imagegeneration module 58 to incorporate noise data 64 and artificial PSFdata 54 into the unblurred test image 60. It is further envisaged thatthe unblurred test image 60 and the blurred test image 66 are receivedas part of a stock of blurred and corresponding unblurred images withoutrequiring artificial generation of blurred images.

In step 220, the blurred input image 24, 66 is deconvolved by theregularized deconvolution sub-module 36 applying the deconvolutionfunction 44, thereby providing the deconvolved image 40. Thedeconvolution function 44 includes the regularization parameter λ, whichis not yet optimized during training, but which is optimally selectedaccording to the training methods described herein during usage.

In step 230, the deconvolved image 40 is received by the convolutionneural network sub-module 38 and passed through the CNN 42 in order todetermine artifacts to be removed. Weights of the CNN 42 are in theprocess of optimization during training and have been optimized andsettled during usage.

In step 240, the deblurred image is output by the non-blind deblurringmodule 34. The CNN 42 generates residuals 48 that are added to thedeconvolved image 40 in order to generate the deblurred output image 20.

Usage step 250 includes controlling a vehicle function, through thevehicle controller 18, based on the deblurred output image 20. Other,non-vehicular, applications are envisaged for the presently disclosednon-blind deblurring technology.

According to step 260, during training, cost function 100 is evaluatedby the cost function module 96 based on a difference between theunblurred test image 60 and the deblurred output image 20. In step 270,the training processing system 50 jointly adjusts weights of the CNN 42and the regularization parameter of the deconvolution function 44 inorder to minimize cost (defined by cost data 98) calculated by the costfunction 100. In embodiments, method 200 includes iteratively adjustingthe regularization parameter λ in steps in order to minimize costs,wherein each iteration includes multiplying back propagated gradientsthrough each layer of the CNN by the derivative of the deconvolved image40 with respect to the regularization parameter λ. The back propagatedgradients represent the derivative of the cost function with respect tothe deconvolved image 40.

According to systems and methods described herein, the regularizationparameter λ can be trained in order to provide enhanced deblurring ofinput images in a processing efficient manner. Exemplary results of thenon-blind deblurring module 34, having CNN weights and theregularization parameter λ determined according to the joint trainingscheme described herein, are shown in FIG. 7. Row 1 shows threedifferent blurred input images A1, B1 and C1 having 2D motion blur.These images are operated upon by the deconvolution function 44(regularized Wiener deconvolution) and its associated trainedregularization parameter λ. The resulting deconvolved images A2, B2 andC2 are shown in row 2. These images are sharper and clearer but includesome artifacts. Ringing artifacts can particularly be seen in image C3.After passing through the CNN 42, clear, sharp and artifact freedeblurred output images A3, B3 and C3 are provided. These output imagescan be used in further processing steps such as for display on a userinterface or in machine vision control functions. Machine vision controlfunctions will be able to better identify features based on thedeblurred output images of row 3, which would not have been practicalbased on the blurred input images of row 1. This is particularly usefulin vehicle control systems where safety can be improved as deblurring isimproved.

It will be appreciated that the disclosed methods, systems, and vehiclesmay vary from those depicted in the Figures and described herein. Forexample, the vehicle 12, the usage and training processing 26, 50 andthe system 10 and/or various components thereof may vary from thatdepicted in FIGS. 1-5 and described in connection therewith. Inaddition, it will be appreciated that certain steps of the method 200may vary from those depicted in FIG. 6. It will similarly be appreciatedthat certain steps of the method described above may occursimultaneously or in a different order than that depicted in FIG. 6.

While at least one exemplary embodiment has been presented in theforegoing detailed description, it should be appreciated that a vastnumber of variations exist. It should also be appreciated that theexemplary embodiment or exemplary embodiments are only examples, and arenot intended to limit the scope, applicability, or configuration of thedisclosure in any way. Rather, the foregoing detailed description willprovide those skilled in the art with a convenient road map forimplementing the exemplary embodiment or exemplary embodiments. Itshould be understood that various changes can be made in the functionand arrangement of elements without departing from the scope of theappended claims and the legal equivalents thereof

What is claimed is:
 1. A method of training a non-blind deblurringmodule, wherein the non-blind deblurring module includes a regularizeddeconvolution sub-module and a convolutional neural network sub-module,the regularized deconvolution sub-module is configured to perform aregularized deconvolution function on a blurred input image to produce adeconvolved image potentially having image artifacts, and theconvolutional neural network sub-module is configured to receive thedeconvolved image as an input to a convolutional neural network and toremove image artifacts, thereby providing a deblurred output image, themethod comprising: receiving, via at least one processor, unblurred testimages and blurred test images, wherein each of the blurred test imagesis related to a corresponding one of the unblurred test images by a blurkernel term and a noise term; jointly training, via the at least oneprocessor, the regularized deconvolution sub-module and theconvolutional neural network by adjusting a regularization parameter ofthe regularized deconvolution function and weights of the convolutionneural network in order to minimize a cost function representative of adifference between each deblurred output image and a corresponding oneof the unblurred test images, thereby providing a trained regularizationparameter, trained weights and a trained non-blind deblurring module;receiving, via the at least one processor, a blurred input image from animaging device; deblurring, via the at least one processor, the blurredinput image using the trained non-blind deblurring module; andoutputting, via the at least one processor, a deblurred output image. 2.The method of claim 1, wherein the deconvolution function is a Wienerdeconvolution function.
 3. The method of claim 1, wherein thedeconvolution function is a Tikhonov-regularized deconvolution function.4. The method of claim 1, comprising using the trained non-blinddeblurring module to deblur the blurred input image, thereby producingthe deblurred output image, wherein the regularized deconvolutionsub-module performs the regularized deconvolution function on theblurred input image to produce a deconvolved image potentially havingimage artifacts, the regularized deconvolution function including thetrained regularization parameter, and the convolutional neural networksub-module processes the deconvolved image through the convolutionalneural network to remove image artifacts, the convolution neural networkincluding the trained weights.
 5. The method of claim 4, wherein theconvolutional neural network outputs residuals and the trained non-blinddeblurring module adds the residuals to the deconvolved image, therebyproducing the deblurred output image.
 6. The method of claim 1, whereinadjusting, via the at least one processor, the regularization parameterand the weights uses a back propagation algorithm.
 7. The method ofclaim 6, wherein the back propagation algorithm adjusts theregularization parameter based on gradients that have been fed back fromthe CNN and a derivative of the deconvolved image potentially havingimage artifacts with respect to the regularization parameter.
 8. Themethod of claim 1, wherein the at least one processor receives theunblurred test images and artificially generates the blurred test imagesusing a blur kernel function and a noise function on the unblurred testimages.
 9. The method of claim 1, wherein the blurred input image isreceived from the imaging device mounted to a vehicle.
 10. The method ofclaim 9, wherein the vehicle includes a vehicle controller and themethod includes controlling at least one vehicle function based on thedeblurred output image.
 11. A system for training a non-blind deblurringmodule, comprising: a non-blind deblurring module including aregularized deconvolution sub-module and a convolutional neural networksub-module, wherein the regularized deconvolution sub-module isconfigured to perform a regularized deconvolution function on a blurredinput image to produce a deconvolved image potentially having imageartifacts, and wherein the convolutional neural network sub-module isconfigured to receive the deconvolved image as an input to aconvolutional neural network and to remove image artifacts, therebyproviding a deblurred output image; an imaging device; and at least oneprocessor configured to execute program instructions, wherein theprogram instructions are configured to cause the at least one processorto: received unblurred test images and blurred test images, wherein eachof the blurred test images is related to a corresponding one of theunblurred test images by a blur kernel term and a noise term; jointlytrain the regularized deconvolution sub-module and the convolutionalneural network by adjusting a regularization parameter of theregularized deconvolution function and weights of the convolution neuralnetwork in order to minimize a cost function representative of adifference between each deblurred output image and a corresponding oneof the unblurred test images, thereby providing a trained regularizationparameter, trained weights and a trained non-blind deblurring module;receive a blurred input image from the imaging device; deblur theblurred input image using the trained non-blind deblurring module; andoutput a deblurred output image.
 12. The system of claim 11, wherein thedeconvolution function is a Wiener deconvolution function.
 13. Thesystem of claim 11, wherein the deconvolution function is aTikhonov-regularized deconvolution function.
 14. The system of claim 11,wherein the trained non-blind deblurring module is configured to deblurthe blurred input image, thereby producing the deblurred output image,wherein the regularized deconvolution sub-module is configured toperform the regularized deconvolution function on the blurred inputimage to produce a deconvolved image potentially having image artifactsusing the trained regularization parameter, and the convolutional neuralnetwork sub-module is configured to process the deconvolved imagethrough the convolutional neural network to remove image artifacts usingthe trained weights.
 15. The system of claim 14, wherein theconvolutional neural network is configured to output residuals and thetrained non-blind deblurring module is configured to add the residualsto the deconvolved image, thereby producing the deblurred output image.16. The system of claim 11, wherein the program instructions areconfigured to cause the at least one processor to adjust theregularization parameter and the weights using a back propagationalgorithm.
 17. The system of claim 11, wherein the program instructionsare configured to cause the at least one processor to adjust theregularization parameter based on gradients that have been fed back fromthe CNN and a derivative of the deconvolved image potentially havingimage artifacts with respect to the regularization parameter.
 18. Thesystem of claim 11, wherein the program instructions are configured tocause the at least one processor to receive the unblurred test imagesand artificially generate the blurred test images using a blur kernelfunction and a noise function on the unblurred test images.
 19. Thesystem of claim 11 comprising a vehicle, the vehicle comprising theimaging device and the non-blind deblurring module, the non-blinddeblurring module configured to receive the blurred input image from theimaging device.
 20. The system of claim 19, wherein the vehicle includesa vehicle controller configured to control at least one vehicle functionbased on the deblurred output image.